01 | Survey

For this first intervention, I contacted a few artists and art historians. Art is subjective, but I’m wondering if it is possible to narrow down descriptions to help machines identify standard features as well as the feelings the artists want to evoke in each art piece. For this first attempt, I listed words that could help me define the things I find essential when describing an artwork.

• Adjectives to describe: composition, light, and colors
• Words (inflections) to define: genre, theme, and feelings

I needed to decide which artworks to use. With so many modern and contemporary artworks, I didn’t know where to begin. I searched for museum collections like the MOMA’s and TATE’s. Although I found MOMA’s database containing all the artworks they acquired since 1929, the list has 15.783 records. I needed to find another collection, closer to where I am, and a place where I could get access. For this, I decided to focus on the MACBA collection. So I went into their webpage and found a vast list of artworks (5.075 records). Still a lot, but I quickly realized that, unfortunately, not all of them had an image, and with only two weeks, I couldn’t lose time waiting for their response. So I centered on the current exhibition and found that they did have a few images uploaded. With these artworks, the questions and the answers. I structured the survey and sent it to friends.

• What genre does this work belong to?
• Concentrate on the “theme.” What is this work about?
• Focus on the “composition.” With what adjectives would you describe this work?
• Center on the “light.” with what adjectives would you describe this work?
• Focus on the “colors,” with what adjectives would you describe this work?
• Concentrate on “feelings.” What do you think the artist wishes to evoke?

With so many modern and contemporary artworks, I didn’t know where to begin. I searched for museum collections like the MOMA’s and TATE’s. Although I found MOMA’s database containing all the artworks they acquired since 1929, the list has 15.783 records. I needed to find another collection, closer to where I am, and a place where I could get access. For this, I decided to focus on the MACBA collection. So I went into their webpage and found a vast list of artworks (5.075 records). Still a lot, but I quickly realized that, unfortunately, not all of them had an image, and with only two weeks, I couldn’t lose time waiting for their response. So I centered on the current exhibition and found that they did have a few images uploaded. With these artworks, the questions and the answers. I structured the survey and sent it to friends.

• What genre does this work belong to?
• Concentrate on the “theme.” What is this work about?
• Focus on the “composition.” With what adjectives would you describe this work?
• Center on the “light.” with what adjectives would you describe this work?
• Focus on the “colors,” with what adjectives would you describe this work?
• Concentrate on “feelings.” What do you think the artist wishes to evoke?

Yes, it probably wasn’t the best idea. I found it interesting how two of them who went to art school together had such different responses. Their conclusions were not the opposite, but they did understand the artworks differently. If they couldn’t agreed on something... How could I create a concise database? & How many responses do I need to trust and catalog an art piece with specific properties?





If art is subjective and guided through personal education, cultural background, and interests, how could a kid, grownup, or blind person understand it equally? Maybe I could train the machine differently.

• Train it to recognize colors and play notes depending on which color the user touches?
• Train it to recognize objects inside the piece? Google has a vast database with drawings of things. Could I use it to identify what is in the artwork? Will it work?
• Train it to get contours out of the pieces to then point out objects? Will people understand it?

02 | Machine Learning

So, I began looking at the basics for building my machine learning, but obviously, this option is going to take me a long, long time as I’m not a developer neither know a lot about coding. I had to find another way to at least try my ideas. Therefore I looked for machine learning projects that I could use. I found the “Teachable Machine” by google and another software called “RunwayML”. I didn’t need any background experience, and I could use pre-trained models as examples.

1. Can “The Teachable Machine” recognize Basquiat and Zush correctly?



Steps:

• Uploading images and training model
• The image uploaded was “Kaenia” by Zush, and the machine decided it was a Basquiat... WRONG!

2. Can the pre-trained model called “denscap” in Runway recognize objects in the artworks?





3. The third model, “photosketch” can define the contours of shapes. But, even when adjusting the parameters, I couldn’t get the desired result.

03 | Modeling & 3D printing

For now... I decided to abstract the artworks and made contours in illustrator to then use Rhino to finally differentiate in a 3D model colors and forms by assigning each a specific height.





For this first experiment, I asked people to describe what they could perceive when touching the 3d model. It was almost impossible. Two of them could identify the lips in Rabascall’s piece; one of them described the horns in Basquiat’s work. Still, nobody could imagine or visualize something else and an urgent description!

Things to take into account if I continue making 3D models:

• Create more difference between heights in model
• Smoothen edges
• Curved surfaces instead of planar (when needed)

General things to take into account: Visual people relate what they see directly to textures. Air can’t be printed. It can’t be solid. What other way can water, air or others can be represented? Temperature? Vibrations? Sound? Other?







For the second try, I asked the woman I’m helping in Fundación Once to do the exact same thing. Although she has very low vision, it was nice to listen to her experience with the model. She highlighted things she would like to appreciate more, like details (stains, rugosity in colors, brushtroke).

General things to take into account: Need to “zoom” on important/ relevant objects or textures to appreciate details.

04 | Where to head

Learn coding (mainly python and javascript) and ML to set “parameters” for translating visual art and test in different ways.

Ideas:

• Sensors translate colors to: vibrations, temperature, sound
• Improve ML code to correctly identify contours for then creating: vibrations, temperature, sound. When touching the lines.
• Train ML to classify brushstroke, composition, light and colors.